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SUMMARY 


The  feasibility  of  space  travel  and  nan— made  satellites 
has  triggered  a  rash  of  interest  in  the  determination  of 
optimal  trajectories  and  generally  in  guidance  and  control 
processes.  These  problems,  for  so  long  of  purely  mathematical 
and  astronomical  concern,  have  now  become  part  of  the 
engineering  domain .  The  result  is  that  there  la  a  great 
demand  for  feasible  numerical  solutions  of  the  associated 
analytic  problems . 

Many  of  these  are  classically  of  great  difficulty.  As  a 
result  of  the  intensive  study  of  these  questions,  it  is  now 
well  appreciated  that  the  classical  techniques  of  the  calculus 
of  variations  are  inoperative  unless  the  problem  Is  rather 
carefully  selected. 

In  this  paper  we  wish  to  sketch  the  applicability  of  a 
new  mathematical  technique,  based  on  the  theory  of  dynamic 
programming,  to  the  computational  solution  of  trajectory 
problems.  Many  problems,  seemingly  inaccessible  to  the 
conventional  methods  of  the  calculus  of  variations,  have 
already  been  resolved. 
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APPLICATIONS  OF  DYNAMIC  PROGRAMMING  TO  SPACE  GUIDANCE, 
SATELLITES,  AND  TRAJECTORIES 

Richard  Bellman 
Stuart  Dreyfus 
Robert  Kalaba 


1.  Introduction 

The  feasibility  of  space  travel  and  man-made  satellites 
has  triggered  a  rash  of  interest  in  the  determination  of 
optimal  trajectories  and  generally  in  guidance  and  control 
processes.  These  problems,  for  so  long  of  purely  mathematical 
and  astronomical  concern,  have  now  become  part  of  the 
engineering  domain.  rIhe  result  is  that  there  is  a  great 
demand  for  feasible  numerical  solutions  of  the  associated 
analytic  problems . 

Many  of  these  are  classically  of  great  difficulty.  As  a 
result  of  the  intensive  study  of  these  questions,  it  is  now 
well  appreciated  that  the  classical  techniques  of  the  calculus 
of  variations  are  inoperative  unless  the  problem  is  rather 
carefully  selected. 

In  this  paper  we  wish  to  sketch  the  applicability  of  a 
new  mathematical  technique,  based  on  the  theory  of  dynamic 
programming  [l]  ,  [2]  ,  to  the  computational  solution  of 
trajectory  problems.  Many  problems,  seemingly  inaccessible 
to  the  conventional  methods  of  the  calculus  of  variations, 
have  already  been  resolved,  [2,4,5,6,10,14]  . 

Naturally,  there  are  still  a  number  of  major  difficulties 
to  be  overcome  before  it  can  be  asserted  that  we  possess  a 
routine  approach  to  realistic  processes.  What  is  important  is 
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that  we  hive  o>  general  method  for  attacking  guidance  and 
control  processes  which  penults  us  to  study  more  complex 
physical  problems  using  quite  elementary  mathematical  concepts. 

2.  A  Property  of  Optimal  Time  Trajectories 

Consider  the  problem  of  going  from  a  point  p  in  phase 
space  to  a  point  q  in  phase  space.  Symbolically,  let  thiB 
be  represented  in  the  figure  below,  where  r  is  am  inter¬ 
mediate  point. 


Let  some  particular  trajectory  be  pursued,  and  denote  by 
t(p,q)  the  tline  required  to  traverse  the  path  between  two 
generic  points.  Then,  clearly 

(1)  t (p,q)  -  t(p,r)  +  t (r,q) . 

Consider  now  the  problem  of  determining,  subject  to 
various  constraints  on  the  allowable  motions,  the  path  of 
minimum  time  between  p  and  q.  Then,  again  clearly,  if  r 
is  a  point  on  an  optimal  trajectory  the  times  between  p  and 
r  and  r  and  q  must  also  be  minimum  times  for  the 
respective  points. 
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The  problem  we  face  Is  that  of  starting  out  from  p.  If 
we  do  not  know  the  optimal  trajectory,  a  priori,  the  fore¬ 
going  bit  of  Information  concerning  geodesics  is  not 
particularly  enlightening. 

We  can,  however,  make  good  use  of  the  following  part  of 
this  Information.  Wherever  r  la  in  phase  space,  and  what¬ 
ever  the  time  consumed  going  from  p  to  r,  the  time 
involved  in  going  from  r  to  q  moat  be  a  minimum  for  this 
part  of  the  path. 

Ttiin  statement  is  a  particular  case  of  the  "principle 
of  optimality"  [l]  . 

3.  Dynamic  Programming  Approach  [3] ,  [4] 

It  remains  to  express  the  foregoing  ideas  in  analytic 
form.  Let 

(1)  f(p)  "  the  time  consumed  in  going  from  an  arbitrary 

point  p  to  a  flx^d  point  q,  using  an 
optimal  p^th . 

Then,  the  principle  of  optimality  yields  firstly  the 
equation 

(2)  f(p)  £  t(p,r)  +  f(r), 

for  any  path  prq.  To  determine  r,  we  minimize  over  all 
admissible  choices  of  r,  obtaining  the  basic  equation 

(3)  f(p)  -  min  [t(p,r)  +  f(r)]. 

r 
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This  equation  1b  the  f'  r  tp._.  computational  solut'on 

of  trajectory  problems. 

Similar  c  ■>nf  Iderat  J  ore  1  >':0  1  ;•  l  io  1  •  - -rmlnat  Ion  of  k— th 
Dost  paths,  as  la  dLsceased  in  [/]  . 


4 .  Braohlstoohrone 

As  an  example,  let  us  consider  the  classical  brachisto¬ 
chrone  problem.  Given  two  points,  P  and  Q,  in  a  vertical 
plane,  we  wish  to  determine  a  curve  connecting  these  two 
points  with  the  property  that  a  particle  sliding  down  this 
curve,  solely  under  the  influence  of  gravity  and  with  no 
frictional  forces,  will  traverse  the  path  in  minimum  time. 

Take  P  to  the  origin  and  lei  0  be  the  fixed  point 
(x0,y0),  as  indicated  below. 

Omitting  the  gravitational  constant,  of  no  import,  the 
problem  in  the  calculus  of  variations  is  that  of  minimizing 
the  integral 


(1) 


J  (y) 


o, 


/?*° 
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Uljt 


,2  1/P 
— )  dx 


over  all  curves  satisfying  the  end  conditions  y(0)  -  0, 
y(xQ)  -  y^.  In  place  of  treating  this  In  the  usual  fashion, 
a  la  Euler  equation  and  :3o  on,  let  ua  Introduce  the  function 
f (x,y)  defined  as  follows: 

(2)  f(x,y)  a  the  time  required  for  a  particle  to  fall 

from  (x,y)  to  (*Q'yo^  along  an  optimal 


path . 
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Then  a  particularization  of  {5.})  yields  the  functional 
equation 


O) 


f(x,y) 


min 

y' 


.1  + 


y’V/2A 

X-.)  a 


+  f  (x  +  A,y  +  y'A)  +  o  (A) 


where  y'  represents  the  slope  of  a  proposed  path  at  x,  and 
A  is  an  infinitesimal. 

Passing  to  the  limit,  we  can  obtain  a  partial  differential 
equation  for  f  which  turns  out  to  be  equivalent  to  the  Euler 
equation.  But  this  is  not  what  we  wish  to  do!  Instead  we  sub¬ 
divide  the  x  interval  into  N  parts,  where  NA  -  x  and 
regard  f(kA,y)  -  f^(y)  as  a  function  of  y  for  0  <  y  <  yQ. 

Then  (2)  becomes 


(M 


My)  - 
k  y' 


■  I  ♦ 


y.2  1/2 


+  fk+i(y  +  y'A) 


for  k  ■  0,1,2,...,N  -  1,  with  f^(y)  9  0.  The  determination 
of  fR(y)  is  a  computational  process  which  we  refer  to  a 
digital  computer.  The  miniraizatlon  over  initial  directions  is 
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carried  out  not  by  means  of  ealculua,  but  by  means  of  a  direct 
searoh  process  over  allowable  slopes. 

One  consequence  of  this  Is  that  constraints  on  y' 
simplify  the  computational  process,  as  they  should  since  we 
have  fewer  policies  to  examine.  Further  details  of  the 
computational  aspects,  whioh  are  seldom  trivial,  will  be 
found  in  [lO]  . 

5 .  Minimum  Energy 

Suppose  that  we  wish  to  go  from  p  to  q  in  phase  space, 
reverting  to  the  general  formulation  of  at  minimum  cost 
in  resources,  say  minimum  fuel.  Regarding  t(p,r)  not  as  a 
time,  but  aa  a  coat  for  getting  from  p  to  r,  we  see  that 
the  general  equation  of  (3.-5)  is  still  valid. 

Oenerally,  we  can  treat  any  variational  problem  in  this 
fashion,  providing  it  has  the  Markovian  property  that  we  have 
been  employing.  To  U3e  the  foregoing  method  we  need  merely 
know  that  the  optimal  continuation  from  any  point  in  phase 
space  depends  only  upon  that  point,  and  not  upon  the  past 
history  of  the  process . 

*Ihua,  not  only  can  general  variational  problems  involving 
standard  functionals  of  the  form 

(l)  J(y)  -  /  1  s(x,y,y’ )dx, 

x0 

be  handled  but  also  many  classes  of  implicit  variational 
problems  [5]  ,  [9J  which  cannot  readily  be  treated  by 
conventional  methods . 
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6.  Stochastic  and  Adaptive  Control  Processes 

Finally,  the  same  general  techniques  can  be  used  to  treat 
the  more  complex  stochastic  and  adaptive  control  processes 
[8,9,11,12]  . 

7.  Discussion 

TT\e  trajectory  and  feedbaok  control  problems  sketched 
in  earlier  sections  represent  merely  one  application,  albeit  a 
particularly  important  one,  of  the  functional  equation  tech¬ 
nique  of  dynamic  programming  to  the  solution  of  space 
problems.  Problems  which  lead  to  multistage  decision  pro¬ 
cesses  also  occur  in  the  areas  of  design  of  multistage  rockets 
[14]  ,  communication  [3]  ,  [l5j  ,  propagation  [l}]  ,  equipment 
reliability  Q.6]  ,  equipment  replacement  [l7]  ,  [l8]  ,  and  so  on. 

The  desire  for  the  design  of  mere  nearly  optimal  systems 
ooupled  with  the  inevitable  advances  in  computer  technology 
should  bring  dynamic  programming  methodology  into  a  position 
of  increasing  importance  in  the  future  in  the  resolution  of 
•paoe  optimization  problems. 


Re  v  . 
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